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ABSTRACT 



At North Dakota State University annual follow-up 



studies have been conducted on teacher education graduates of the 
secondary teacher preparation program since 1963. Each year, survey 
forms are sent to all teacher education graduates of the previous 
year and to the principals or superintendents of graduates who are 
teaching. The purpose of the follow-up studies is program evaluation 
focusing on accountability and program improvement. Results of the 
surveys are considered in decision making about curriculum 
development. In addition to the annual graduate and administrator 
surveys, North Dakota State University has conducted or participated 
in other national and institutional follow-up activities. Although 
not all recommendations from the follow-up studies result in 
curriculum changes, some recommendations have initiated discussion 
and eventual change. Some of these changes have been in the areas of 
classroom management, evaluating student learning, and multicultural 
education. Faculty development has been provided, and data from 
follow-up studies have been used in program planning and revision. 
Examples of the survey forms sent to graduates and supervisors are 
included. (I AH) 
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Follow-up Studies in Program Evaluation 
Hummell and Strom (1S87) attribute increased use of follow-up studies to institutional 
demands for accountability and to accreditation demands. At North Dakota State University 
(NDSU), formal follow-up studies have been carried out annually, at least since 1963. For us, yes, 
they are worth the trouble. 

At NDSU approximately 100 secondary teachers are prepared each year. We prepare no 
elementary teachers. Secondary teachers are prepared in agricultural education, English, history, 
home economics, mathematics, modem languages, music, physical education, sciences, social 
sciences, and speech. We send out a survey form to all the teacher education graduates of the 
previous year. We also send a form to the principal or superintendent of all the graduates who are 
teaching-now first-year teachers. Data are analyzed separately for those teaching and those not 
teaching. 

The purpose of the follow-up study is program evaluation, to make judgments about the 
worth of the program. Although Galluzzo and Craig (1990) describe four purposes for teacher 
education program evaluation (p. 605), we focus on two-improvement and accountability. Whether 
or not educators wish to use the term "accoun .ability," they are invariably involved in curriculum and 
instructional decision maKing, And to make these decisions, program evaluations are necessary. 

While the debate has continued over what is, or ought to be, acceptable as program 
evaluation (see, for example, Worthen & Sanders, 1973, pp. 17-26, or Brinkerhoff, Brethower, 
Hluchyj, & Nowakowski, 1983, pp. xiv-xx), the view of program evaluation at NDSU has been that 
the purpose is to make judgments about the worth of the program as a part of an overall curriculum 
development process. 

There are four major components to program development: (a) identifying the program 
goals and objectives (what is necessary for a beginning teacher), (b) selecting the means for 
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attaining the goals and objectives (courses, activities, experiences), (c) organizing these means 
(sequence, instructional methods), and (d) evaluating the outcomes. These four are 
interdependent. The evaluation of each of these four components is continuous and each is 
affected by all of the others. Decisions on the selection of educational objectives, selection and 
organization of subject matter, organization of instructional methods and learning experiences, and 
use of systematic evaluation procedures are philosophically based (Tanner & Tanner, 1 980). At 
NDSU the philosophical base is our view of what a beginning secondary teacher needs to know 
and bs able to do. The program evaluation needs to reflect the interconnectedness of all of these. 

Faculty, with input from students, practitioners, and advisory committee members, design the 
program. They plan the program goals and objectives to reflect what a beginning teacher should 
know and be able to do, design courses to provide the content and opportunities for learning 
activities to meet the objectives, organize and sequence the activities, and design the program 
evaluation to see how well the program is accomplishing the goals. The program evaluation also 
examines the appropriateness of the goals and objectives, courses and content, learning activities 
and experiences, and the evaluation process, as well in identifying data that will provide evidence 
of effectiveness. 

When a program is defined to include antecedents or preexisting conditions, such as the 
characteristics of the students (set by the program's admission requirements), the processes of tht 
program (learning activities and experiences), and the outcomes, then follow-up studies can 
contribute meaningful data related to program outcomes or effects. Follow-up studies may also help 
to identify unintended outcomes. First-year teacher comments to the open-ended questions may 
point out any unintended outcomes. 

Since one of the goals at NDSU is program improvement, program participants, both faculty 
and former students, play an important role in providing feedback. Information from follow-up 
studies is used by program faculty to make adjustments in the program. The input from employers 
(principals and superintendents) of the first-year teachers is also used. 
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Surv ey Form 

A survey form is used to solicit input from the graduates of the teacher education program. 
The form is designed to reflect the objectives of the program. There are 26 items to be rated using a 
five-point Likort scale from Very Effective to Very Ineffective. A copy of the current form (pink) is 
attached (page 9). We fill in the former student's name and major before we send it out. 

As the program has been revised, so has the survey form. For example, NDSU had a 
Dean's Grant prelect from 1980 to 1983 to prepare secondary teachers for mainstreamed learners 
in their classrooms. We added items to the survey form during the first year of the project although 
the graduates who would receive the form had not had tha planned instruction. We did the same 
thing when the state mandated that coursework on North Dakota Native Americans be added to the 
teacher preparation program. We added items to the survey form before the graduates could be 
expected to have received the instruction. When changes are made in the program, there should 
be changes in the evaluation ratings. If the changes in the instructional program were effective, the 
ratings by the graduates in those areas would improve. 

When the faculty completed their identification of the theme, model, and knowledge bases 
undergirding the program for the NCATE visit under the redesigned standards, the survey form was 
revised to reflect those objectives. At that time, the entire undergraduate curriculum was changed 
and a common core for all secondary teachers was implemented in September, 1 987. 

Since the context of the teaching position is important, the first-year teachers are asked 
(back side of form) to describe students in their classrooms on two dimensions-racial diversity and 
handicapping condition. In addition, there are open-ended questions about their perceptions of the 
strengths of the program and ways to improve the program in the areas of the teaching specialty 
and teaching skills. 

In April, the survey fomi is mailed to all teacher education graduates of the previous year. A 
similar survey form is sent to the administrators of all graduates who are teaching (their first 
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year as a teacher). A copy of the current form (beige) is attached (page 11). We fill in the first-year 
teacher's name and major before we send it. 

Data are analyzed using a computer program. Data are kept separate for those teaching 
and those not teaching. Data are reported for the entire group, and by the ' jhing specialty. (All 
names are removed.) A report is prepared by the Dean's Office including the summary data, 
comparison of data to previous year(s), and interpretations of the results. The report also includes 
the data from the school administrators. All comments written in by graduates (teaching and not 
teaching) and by administrators are typed as an appendix of the report. The report is distributed to 
the School of Education Curriculum Committee and to teacher education faculty. 
Results 

About half of those prepared to teach are employed as teachers at the time the survey is sent 
out (April of the year following graduation). The response rate on the survey ranges from 72%-82%. 
The response rate for the administrators (as the employers of the graduates) is higher, 84%-9G%. 
The administrators also rate the teachers' effectiveness as higher than the first-year teachers rate 
the effectiveness of their preparation. 
Other Follow-up Activities 

In the spring of 1989, NDSU participated in a national data collection effort spearheaded by 
the National Center for Research on Teacher Education at Michigan State University (Freeman, 
1989). The 1987-88 graduates were sent the survey. No form was sent to administrators. The 
items on the form were developed together with the participating institutions. In an attempt to meet 
everyone's needs, the survey form was longer and asked questions related to more topics. Our 
response rAe that year was 64.3%. We had comments from some of our graduates that the form 
"didn't apply" to them. For example, there were items for elementary teachers, We are glad we 
participated and felt it was worthwhile. We agree with Loadman (1989) that linkages can be 
productive in program evaluation efforts. There is now a network developing for sharing evaluation 



practice and data. We agree that is valuable. We have, however, continued our own survey form. It 
is improved as a result of our participation in the national data collection project, 'or example, better 
items. We feel the need for more specific feedback related to our program. The national survey was 
not particularly useful in tracking program changes. 

In 1987, telephone interviews were conducted with the graduates who were teaching. 
Questions focused on the effectiveness of their preparation in relation to their jobs as first-year 
teachers. In addition, they were asked if there were ways in which faculty or others could assist 
them. The data did not provide additional information beyond the survey so the interviews have not 
been continued. It did help the first-year teachers feel important. 

In 1986, an additional sheet was added to both the first-year teacher and the administrator 
form, it sought information from both as to the kind of help desired in a beginning teacher 
assistance program. These data provided one basis for a funding proposal to develop an 
assistance program. 

Much earlier, a study was carried out on concerns of first-year teachers rr *d to situational 
factors (Lundstrom & Murphy, 1976). In this study, the first-year teachers received a letter each 
month telling them we cared about how they were doing and encouraging their participation in 
returning a card responding to an incomplete sentence, "The way I feel about teaching this - jnth 
is ... " Content analysis was used to categorize responses as positive, negative, or neutral. 
Further, the response was categorized as to the topic or concern expressed, such as work load, 
administration, students, teaching in general, teacher's own capabilities, colleagues, or parents. In 
addition to contributing to our knowledge about the concerns of first-year teachers, it was an 
effective kind of follow-up study. The first-year teachers also loved all the attention! (At that time, 
most of them were teaching in very rural, isolated areas of North Dakota.) 
Uses of the Follow-up Study Data 

Recommendations, ratings, and comments from follow-up studies do not, by themselves, 
result in changes in the program. Multiple data sources are used. When the administrators report 



the same things as the first-year teachers, more attention is given and change is more likely. 
However, recommendations fo>' changes in the program from follow-up study data have been 
implemented As an example, ratings on managing learner behavior were low in the follow-up 
studies, especially in some teaching fie'ds. There was also some anecdotal evidence from some 
classroom teachers that some of our first-year teachers were having some trouble with "discipline." 
These things (along with others) were considered and in the creation of the common core 
curriculum, a course on classroom management was added. Since *hen, ratings on this item have 
gradually increased. 

Reported feelings by first-year teachers with regard to evaluating student learning (also 
supported by administrator comments and requests for help from faculty by some beginning 
teachers) resulted in a required course in appraising student learning in the new common core. It 
previously had been required only in some teaching specializations. 

Multicultural education and teaching mainstreamed learners in regular classrooms are 
infused in our program, not offered as separate courses. Comments from first-year teachers (again 
supported by administrators) have resulted in increased instructional time and strategies in these 
two areas. Again, ratings have gradually improved. 

In the last five years, the teaching assignments of several faculty have been changed (not 
solely based on follow-up study data, but it had an effect). Conferences have been held and faculty 
development provided for some faculty to address concerns raised in the follow-up studies. Not all 
changes have related to the curriculum. 

Data from follow-up studies (along with other inputs) have been used in program planning 
and revision. Now that we have in place a common program to prepare secondary teachers, the 
data from the follow-up studies have more meaning. Faculty can no longer say, "Well, that doesn't 
apply to music," or "... to history." Since the professional education component is common to all, 
weakness in that preparation applies to all. As mentioned in the Katz, Raths, Mohanty, Kurachi, and 
Irving (1981) article, faculty could synthesize the curriculum in a new way (which they need at 



NDSU in the creation of the common core) and it would likely, according to Katz et al ( 'iiW, "give 

rise to inter-faculty contention" (which it did). 

Conclusion 

In conclusion, at NDSU, we do report data for those teaching separately; we include 
employers (principals or superintendents), and the focus is on program evaluation, not employment 
information. Data from the follow-up studies provide baseline information on our first-year teachers. 
We find follow-up studies an excellent source of data, not the ratings per se, but the change in the 
ratings over time (the three years it takes to complete the program). They have had an impact on 
the program, its direction and its development. 

Program development through program evaluation is enhanced with the defining of what it 
means to be a beginning teacher, how the beginning teacher is to be prepared, and what data are 
indicators of effective preparation (Galluzzo & Craig, 1990, p. 612). Inclusion of follow-up data in 
the mix for program decision making is a worthy goal. 

Since most of our teacher education graduates leave the state, classroom visits are not 
practical. (They accept teaching positions in states where salaries for beginners are significantly 
higher.) While follow-up studies are not perfect, they are about the best we have, and we feel they 
are worth the effort. 
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V4UV U ULt J. ^ \J 

GMDUATES 



TEACHER EDUCATION PREPARATION 

North Dakota State University 
School of Education 



Do nor mar* It 

□ □□□ 



sOD 



Name _ 

Name of School 
Subjects Taught 



.Major 
.Minor 
.Date 



13 
15 
17 
19 



□ [— I Dlr»ctlon»r/=or each of the items below, circle the number which reflects your evaluation or 
LJ attitude regarding your preservice preparation. Circla only one number for each Hem. > 

□ □ 

□ □ 

□ □ 



s-vyy EM •ctfwi 

4 Cfftctlvs 

3 Sofffewflft Effective 



1**"V#fy bieffecflvs 



q g 1 • Evaluate the effectiveness of your preservice preparation in each of the following areas: 

I I 



21 



25 



26 



27 



9 

ERIC 



Planning instruction 

n a. Planning and writing lessons 
(e.g., flexibility in * cavities, clarrty of 
plans, time allocation for activities) 

22 Q b. Developing clearly stated instruc- 

tional objectives 

23 O c. Putting subject matter in a se- 

quential order 

24 Q d. Designing and organizing in- 

structional activities to enhance 
learning 

I I e. Selecting or developing instruc- 
tional matenals or media to 
enhance learning 

Implementing Instruction 

I I f. Applying learning theory and 
principles of learning 

Q g. Managing learning behavior 
(e.g., reinforcing appropnate 
behavior, preventing misbehavior, 
controlling misbehavior, and 
discipline) 

[ j h. Using a variety of instructional 
techniques to accommodate dif- 
ferences in learning styles and 
abilities among stur ents 

f"| i. Understanding vaiious customs, 
values, and diverse cultural 
backgrounds of students 

f"1 j. Demonstrating sensitivity to con- 
temporary democratic issues such 
as racism and sexism (e.g., treating 
all students equrtabfy, selecting un- 
biased resource?, and using 
nonstereotypical language) 

I I Ic Using community resources to 
enhance student learning 

Implementing Evaluation 

t~1 I. Designing or selecting valid and 
reliable evaluation instruments 
suitable to instructional objectives 
and the conceptual level of the 

student- — — — ' ~ 



28 



29 



30 



31 



32 



ill 



Mill 

5 4 3 2 1 
5 4 3 2 1 
5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



33 LJ m. Conducting and using observa- 

tion techniques to evaluate student 
progress or student behavior (e.g., 
the use of rating scales, checklists, 
or anecdotal records) 

34 r Zl n. Using evaluation for making in- 

structional deosons (e.g., for 
review, grouping of students, 
remedial work, placement) 

35 LJ °- Using a variety of evaluation 

devices or activities for progress 
reports, feedback, or grading 

38 p. Evaluating stypJert growth on a 
continuous, systematic basis 

Developing Professional 
Behavior 

37 O q. Evaluating one's own instruc- 

tional skMs through gathering, inter* • 
p'eting, and using data tor 
^;NmprovemerK 

38 Q r - Establishing and maintaining ef- 

fective worwng relationships with 
colleagues and other individuals 
encountered in professional 
situations 

30 s. Encouraging students' feelings of 
self-worth 

40 CD Accenting constructive criticism 

n - » ; ^ and •> c rjggesbons for profes* * 
aior' .^prr.vement 

41 Q u - ^0P*ri3 and maintaining ef- 

fect teacher/student relationships 

42 CH v « Developing skHI in verbal and 

written communication 

43 D vv. Understanding the value of 

r#fletf ng a professional 
appearance 

44 (ZI * Understanding the value of par- 

ticipating in praesstonaJ groups or 
activities 

45 CI y- Developing krtovriedge of the 

subject matter 



r 

> M 



r 

> 



5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 

5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5~4 3 



^^fn^^i^S^ ****** wprwj**** 

\ 2 1 and the program 5 4 3 2 1 
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2. How many students in your classes are classified into eacn of the following groups? (Pli 
not percentage or words.) 



indicate u number, 



*>□□□ 

«□□□ 
«□□□ 



«•□□□ 



Racist groups: 



.Asian 

.Black/Negro 
J-lispanic 
Jslauve American 
.Caucasian 



Handicapping conditions: 



Emotionally disturbed 
Xeanng impaired 
" lentally handicapped 
.Orthopedically or health impaired 
JSpeech impaired 
.Visually handicapped 
.Learning disabled 



3. Based upon your professional experience, in what ways could we improve our program regarding: 
a. subject matter specialty? 



b. teaching skills? 



4. What do you consider to be the strengths ot the teacner education preparation (e.g., teaching methods, student 
teacnmg) in your unaergraouate program? 



5. Please make any comments on- any items not previously included. 
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i □□□□ 
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Name of Teacher 
Name of School . 
Subjects Taught _ 



• □□ 
iDU 

• □□ 
.1 □ □ 
»□□ 
uC □ 

17 □ □ 

Planning Instruction 

□ 



Name of Respondent. 
Title of Respondent . 
Date_ 



Directions: For each of th& items below, circle the number which reflects your evaluation 
or attitude regarding the teacher. Circle only one numbar lor each Item. The information 
you provide will be uaed to evaluate the teacher education program rather than the 
teacher. 

S— Vtry Eflvcthm 
4— Ettsettvs 
3— Sonwwtttt Eftecttv* 
2— iMftectlvs 
1— Vtry liMfftcttv* 

1 . For each of the following, evaluate the effectiveness of the teacher's: 

I 



21 



22 



23 



24 



25 



□ 
□ 
□ 

□ 



26 



□ 

I ! 

27 l_J 



28 



29 



30 



31 



32 



a Planning and writing lessons 
(e.c flexibility in activities, clarity of 
plans, time allocation for activities) 

b. Developing clearly stated instruc- 
tional objectives 

c. Putting subject matter in a se- 
quential order 

d. Designing and organizing in- 
structional activities to enhance 
learning 

e. Selecting or developing instruc- 
tional materials or media to 
enhance learning 

Implementing Instruction 

f Applying learning theory and 
principles of learning 

g Managing learning behavior 
(e.g., reinforcing appropriate 
behavior, preventing misbehavior, 
controlling misbehavior, and 
discipline) 

Q h Using a variety of instructional 
techniques to accommodate dif- 
ferences m learning styles and 
abilities among students 

| | i, Understanding various customs, 
values, and diverse cultural 
backgrounds of students 

1 | ) Demonstrating sensitivity to con- 
temporary democratic issues such 
as racism and sexism (e.g., treating 
all students equitably, selecting un- 
biased resources, and using 
nonstereotypical language) 

[ ] k. Using communrty resources to 
enhance student learning 

Implementing Evaluation 

f~] l, Desipmng or selecting valid and 
reliable evaluation instruments 
suitable to instructional objectives 
and the conceptual level of the 



i I 

r o 



5 4 3 2 1 



I 
I 

r 

> 



5 4 3 2 



5 4 3 2 1 



5 4 3 2 1 



4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



33 



34 



35 



36 



37 



38 



39 



40 



□ 

□ 

□ 
□ 

□ 
□ 

□ 
□ 



41 



42 



43 



□ 
□ 
□ 

5 4 3 2 1 44 □ 

□ 
□ 



45 



4$ 



m. Conducting and using observa- 
tion technique* to evaluate student 
progres or student behavior (e.g., 
the use of rating scales, checklists, 
or anecdotal records) 

n. Using evaiua *on for making in- 
structional decis vis (e.g., for 
review, grouping ot stuoents, 
remedial work, placement) 

o. Using a variety of evaluation 
devices or activities for progress 
reports, feedback, or grading 

p. Evaluating student growth on a 
continuous, systematic basis 

Developing Professional 
Behavior 

q. Evaluating one's own instruc- 
tional skills through gathenng, inter- 
preting, and using data tor 
self-improvement 

r. Establishing and maintaining ef- 
fective working relationships with 
colleagues and other individuals 
encountered in professional 
situations 

s. Encouraging students' feelings of 
self-worth 

t. Accepting constructive criticism 
and using suggestions tor profes- 
sional improvement 

u. Developing and maintaining ef- 
fective teacher/student relationships 

v. Exhibiting skill in verbal and writ- 
ten communication 

w. Reflecting a professional 
appearance 

x. Participating in professional 
groups or activities 

y. Demonstrating knowledge of the 
subject matter 

z. Managing materials, facilities, 
and the program 



I 
i 



i i ! f 

> m m £ 

5 4 3 2 



5 

m 

> 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



5 4 3 2 1 



i : 



2. To prepare a better teacner, in wnat ways could we improve our program regarding: 
a. suoject matter specialty? 



b. teacning skills? 



3. Please maKe general comments on any items not included above. 
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